Viewpoint-Controllable Telepresence: A Robotic-Arm-Based Mixed-Reality Telecollaboration System.

In mixed-reality (MR) telecollaboration, the local environment is remotely presented to a remote user wearing a virtual reality (VR) head-mounted display (HMD) via a video capture device. However, remote users frequently face challenges in naturally and actively manipulating their viewpoints. In this paper, we propose a telepresence system with viewpoint control, which involves a robotic arm equipped with a stereo camera in the local environment. This system enables remote users to actively and flexibly observe the local environment by moving their heads to manipulate the robotic arm. Additionally, to solve the problem of the limited field of view of the stereo camera and limited movement range of the robotic arm, we propose a 3D reconstruction method combined with a stereo video field-of-view enhancement technique to guide remote users to move within the movement range of the robotic arm and provide them with a larger range of local environment perception. Finally, a mixed-reality telecollaboration prototype was built, and two user studies were conducted to evaluate the overall system. User study A evaluated the interaction efficiency, system usability, workload, copresence, and user satisfaction of our system from the remote user's perspective, and the results showed that our system can effectively improve the interaction efficiency while achieving a better user experience than two traditional view-sharing techniques based on 360 video and based on the local user's first-person view. User study B evaluated our MR telecollaboration system prototype from both the remote-user side and the local-user side as a whole, providing directions and suggestions for the subsequent design and improvement of our mixed-reality telecollaboration system.

Research on face specular removal and intrinsic decomposition based on polarization characteristics.

It is well known that the specular component in the face image destroys the true informantion of the original image and is detrimental to the feature extraction and subsequent processing. However, in many face image processing tasks based on Deep Learning methods, the lack of effective datasets and methods has led researchers to routinely neglect the specular removal process. To solve this problem, we formed the first high-resolution Asian Face Specular-Diffuse-Image-Material (FaceSDIM) dataset based on polarization characterisitics, which consists of real human face specular images, diffuse images, and various corresponding material maps. Secondly, we proposed a joint specular removal and intrinsic decomposition multi-task GAN to generate a de-specular image, normal map, albedo map, residue map and visibility map from a single face image, and also further verified that the prediected de-specular images have a positive enhancement effect on face intrinsic decomposition. Compared with the SOTA algorithm, our method achieves optimal performance both in corrected linear images and in uncorrected wild images of faces.

Design of a near-eye display measurement system using an anthropomorphic vision imaging method.

We developed a new near-eye display measurement system using anthropomorphic vision imaging to measure the key parameters of near-eye displays, including field-of-view (FOV), angular resolution, eye box, and virtual image depth. The characteristics of the human eye, such as pupil position, pupil size variation, accommodation function, and the high resolution of the fovea, are imitated by the proposed measurement system. A FOV scanning structure, together with a non-vignetting image-telecentric lens system, captures the virtual image from the near-eye display by imitating human eye function. As a proof-of-concept, a prototype device was used to obtain large-range, high-resolution measurements for key parameters of near-eye displays.

Method for reconstructing a high dynamic range image based on a single-shot filtered low dynamic range image.

Traditional cameras are limited by sensors and cannot directly capture single-shot high dynamic range (HDR) images. We propose an improved single-shot HDR image reconstruction method that uses a single-exposure filtered low dynamic range (FLDR) image. First, by adding an optical filter in front of the camera lens, a FLDR image with different RGB channel exposure states and luminance ranges can be captured in a single-shot, unlike the traditional LDR image. Second, a deep inverse tone mapping network (DITMnet) with multibranch features extraction and multioutput images synthesis is designed to reconstruct an HDR image from a single FLDR image. Experimentally, under different exposure states and color spaces, our method outperforms similar algorithms.

HiFinger: One-Handed Text Entry Technique for Virtual Environments Based on Touches between Fingers.

We present a text entry technique called HiFinger, which is an eyes-free, one-handed wearable text entry technique for immersive virtual environments by thumb-to-fingers touch. This technique enables users to input text quickly, accurately, and comfortably with the sense of touch and a two-step input mode. It is especially suitable for mobile scenarios where users need to move (such as walking) in virtual environments. Various input signals can be triggered by moving the thumb towards ultra-thin pressure sensors placed on other fingers. After acquiring the comfort range of the touch between the thumb and other fingers, six placement modes for text entry are designed and tested, resulting in an optimal placement mode that leverages six pressure sensors for the text entry and two for the control function. A three-day study is conducted to evaluate the proposed technique, and experimental results show that novices can achieve an average text entry efficiency of 9.82 words per minute (WPM) in virtual environments based on head-mounted displays after a training period of 25 min.

Design and assessment of a 360° panoramic and high-performance capture system with two tiled catadioptric imaging channels.

It has always been a challenge to obtain a 360° panoramic imaging system with a high-resolution and non-visible seam. In this study, a 360° panoramic and high-performance system has been developed based on two tiled imaging channels with a field of view of 190° and F/# of 2.0. In this study, some issues regarding 360° panoramic tiled methods, such as relative illuminance and ghost images, were analyzed in detail and taken into consideration during the design procedure. In the final design, the stop has been fulfilled for each field angle, and the ghost images have also been well optimized. A proof-of-concept prototype producing a high-performance 360° panoramic video has been developed.

An Improved Method of Pose Estimation for Lighthouse Base Station Extension.

In 2015, HTC and Valve launched a virtual reality headset empowered with Lighthouse, the cutting-edge space positioning technology. Although Lighthouse is superior in terms of accuracy, latency and refresh rate, its algorithms do not support base station expansion, and is flawed concerning occlusion in moving targets, that is, it is unable to calculate their poses with a small set of sensors, resulting in the loss of optical tracking data. In view of these problems, this paper proposes an improved pose estimation algorithm for cases where occlusion is involved. Our algorithm calculates the pose of a given object with a unified dataset comprising of inputs from sensors recognized by all base stations, as long as three or more sensors detect a signal in total, no matter from which base station. To verify our algorithm, HTC official base stations and autonomous developed receivers are used for prototyping. The experiment result shows that our pose calculation algorithm can achieve precise positioning when a few sensors detect the signal.